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Research has shown that teacher effectiveness is the most 
important school-based factor that influences student 
achievement.’ Studies have also shown that there is 
substantial variation in teacher effectiveness to improve 
student outcomes.’ Literacy Design Collaborative aims to help 
teachers improve their effectiveness in the classroom with a 
focus on supporting their literacy instruction. Literacy Design 
Collaborative provides professional development, coaching, 
and resources to support teachers to work collaboratively 

in their schools to create and use high-quality literacy 
instruction materials aimed at improving students’ reading, 
research, and writing skills. Teachers across content areas— 
including English language arts, social studies, and science— 
can use the Literacy Design Collaborative program.* 


This What Works Clearinghouse (WWC) report, part of the 
WWC’s Teacher Excellence topic area, explores the effects 
of the Literacy Design Collaborative program on student 
achievement. The WWC identified five studies of Literacy 
Design Collaborative. Three of these studies meet WWC 
standards. The evidence presented in this report is from 
three studies of the effects of Literacy Design Collaborative 
on racially, ethnically, and geographically diverse student 
samples—in grades 4 through 8—in schools in Kentucky, 
New York City, and a school district on the West Coast. 


What Happens When Teachers Participate in Literacy Design Collaborative?* 


The evidence indicates that implementing Literacy Design 
Collaborative: 


e Has inconsistent effects on general literacy 
achievement. 

e May result in little or no change in general social 
studies achievement. 


Findings on Literacy Design Collaborative from three studies 
that meet WWC standards are shown in Table 1. The table 


reports an effectiveness rating, the improvement index, and 
the number of studies and students that contributed to the 
findings. The improvement index is a measure of the inter- 
vention’s effect on an outcome. It can be interpreted as the 
expected change in percentile rank for an average comparison 
group student if that student had received the intervention. 


The evidence presented in this report is based on available 
research. Findings and conclusions could change as new 
research becomes available. 


Table 1. Summary of findings on Literacy Design Collaborative from studies that meet WWC standards 


Study Findings 


Evidence meeting WWC standards (version 4.0) 


Improvement index 


Outcome domain Effectiveness rating (percentile points) Number of studies Number of students 
General literacy achievement Mixed effects +{ 3 31,003 
General social studies achievement No discernible effects 0 1 19,962 


Note: The improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the 
intervention. For example, an improvement index of +1 means that the expected percentile rank of the average comparison group student would increase by 1 point if the 
student received instruction from a teacher who used Literacy Design Collaborative. For general literacy achievement, the improvement index values are generated by 
averaging findings from the outcome analyses that meet WWC standards, as reported by Herman et al. (2015), Wang et al. (2018), and Wang et al. (2020). For general 


social studies achievement, the improvement index values are generated by the one analysis conducted within the domain, as reported by Herman et al. (2015). A positive or 
negative improvement index does not necessarily mean the estimated effect is statistically significant. General literacy achievement outcomes reported in these studies include 
the Kentucky Performance Rating for Educational Progress (K-PREP) tests in reading and writing, the New York State English Language Arts Assessment, and the Smarter 
Balanced English Language Arts Assessment used in a West Coast school district. The general social studies achievement outcome is the K-PREP social studies assessment. 
The effects of Literacy Design Collaborative are not known for other outcomes within the Teacher Excellence topic area, including general mathematics achievement, general 
science achievement, general achievement, English language proficiency, staying in school, progression in school, completing school, student social interaction, observed 
individual behavior, student emotional status, student engagement in school, instructional practice, teacher attendance, teacher retention at the school, teacher retention in the 
school district, teacher retention in the state, or teacher retention in the profession. 


BOX 1. HOW THE WWC REVIEWS AND DESCRIBES EVIDENCE 


The WWC evaluates evidence based on the quality and results of reviewed studies. The criteria the WWC uses for evaluating 
evidence are defined in the Procedures and Standards Handbooks and the Review Protocols. The studies summarized in this report 
were reviewed under WWC Standards (version 4.0) and the Teacher Excellence topic area protocol (version 4.0). 


To determine the effectiveness rating, the WWC considers what methods each study used, the direction of the effects, and the 
number of studies that tested the intervention. The higher the effectiveness rating, the more certain the WWC is about the reported 
results and about what will happen if the same intervention is implemented again. The following key explains the relationship between 
effectiveness ratings and the statements used in this report: 


Effectiveness Rating 


Rating interpretation 


Description of the evidence 


Positive (or negative) effects 
outcome 


The intervention is likely to change an 


Strong evidence of a positive (or negative) 
effect, with no overriding contrary evidence 


Potentially positive (or negative) effects 


The intervention may change an outcome 


Evidence of a positive (or negative) effect 
with no overriding contrary evidence 


No discernible effects 


The intervention may result in little to no 
change in an outcome 


No affirmative evidence of effects 


Mixed effects 
onan outcome 


The intervention has inconsistent effects 


Evidence includes studies in at least two of 
these categories: studies with positive effects, 
studies with negative effects, or more studies 
with indeterminate effects than with positive or 
negative effects 


How is Literacy Design Collaborative Implemented? 


The following section provides details of how districts and 
schools implemented the Literacy Design Collaborative 
program. This information can help educators identify 
the requirements for implementing the Literacy Design 
Collaborative and determine whether implementing this 
intervention would be feasible in their districts or schools. 
Information on Literacy Design Collaborative presented 

in this section comes from the studies that meet WWC 
standards and from correspondence with the developer. 


¢ Goal: Literacy Design Collaborative aims to help teachers 
develop and use high-quality, standards-aligned literacy 
instructional materials that help build students’ reading, 
research, and writing skills. 


¢ Target population: Literacy Design Collaborative is 
designed to support teachers in kindergarten through 
grade 12 across subject areas. 


¢ Method of delivery: Materials are provided online to 
support teachers in adapting or developing standards- 
aligned literacy tasks. In addition, schools provide 
collaborative planning time in a professional learning 
community, and trained Literacy Design Collaborative 
coaches provide virtual support and feedback. 
Participating teachers do not have to travel for the 
program though they may attend an in-district orientation 
for the program in the summer. School and district 
instructional leaders, including a teacher-leader, may 
attend virtual and in-person coaching institutes offered by 
Literacy Design Collaborative. 


¢ Frequency and duration of service: Teachers develop 
and implement between one and four 2- to 3-week 


Comparison group: In the three studies that 
contribute to this intervention report, students in 
the comparison group were taught by teachers who 


did not participate in Literacy Design Collaborative. 
Teachers may have participated in other training or 
professional development programs offered by their 
schools or school districts. 


instructional modules over the course of a school year. 
Teachers participate in at least 45 minutes of collaborative 
planning time every week in a professional learning 
community. Every other week, the planning time includes 
virtual coaching from Literacy Design Collaborative 
coaches and time to engage in online course sessions. 

The amount of planning and coaching time varied across 
the studies that contribute to this report. In Herman et 

al. (2015), teachers were expected to participate in at 
least two professional development sessions—including 
classroom visits, coaching, meetings, or online courses— 
during the school year. Teachers in the other two studies 
(Wang et al., 2018, 2020) also received classroom visits 
and coaching, but formal professional development 
meetings were not listed among the activities occurring in 
those studies. Refer to Table 2 for additional details. 


Intervention components: The key components of 
Literacy Design Collaborative—including the instructional 
modules, student assessment, collaboration in 
professional learning communities, and coaching—are 
described in Table 2. 


Table 2. Components of the Literacy Design Collaborative program 


Key component 


Description 


Instructional modules 
and tasks 


Rubrics to assess 
student work 


Professional learning 
community 


Remote coaching 


Leadership support 


Teachers work collaboratively and with support from coaches to develop new or implement existing instructional modules 
and tasks aimed at building students’ reading, research, and writing skills. The modules are 2- to 3-week instructional plans 
for teaching literacy that are aligned with Common Core State Standards and address a particular theme or content area like 
English language arts, social studies, or science. At the end of each module, students complete a culminating writing task. 
Teachers implement one to four instructional modules over the course of a school year. Modules may also include mini-tasks, 
which are short literacy tasks that may occur during a single class period. 


To develop the modules and tasks, teachers use a framework created by the Literacy Design Collaborative that guides 
teachers in using fill-in-the-blank templates available in an online portal called CoreTools and illustrates the requirements for 
designing Common Core-aligned assignments using the templates. Teachers can access vetted literacy modules, tasks, and 
other resources through CoreTools. When choosing from the instructional modules and tasks on CoreTools, teachers can 
review assessments of the resources from other teachers who have used it in their classrooms, including descriptions of the 
quality, coherence, and alignment to Common Core standards. 


Nearly all teachers in Herman et al. (2015) taught two instructional modules related to Literacy Design Collaborative during 
the school year. The authors do not describe the implementation experience of the sample of teachers in the other two studies 
that contribute to this intervention report (Wang et al., 2018, 2020). 


Literacy Design Collaborative provides five rubrics that teachers use to assess the disciplinary content of student work 

on the culminating writing task and whether the work meets literacy standards. The five rubrics—argumentative writing; 
informational writing; Next Generation Science Standards; National Council for the Social Studies College, Career, and Civic 
Life Framework for Social Studies State Standards; and Common Core State Standards for reading—were designed by the 
Stanford Center for Assessment Learning and Equity. 


Schools implementing the Literacy Design Collaborative program create professional learning communities by reserving at 
least 45 minutes every week for common planning time for teachers to collaborate on instructional modules and tasks. A 
teacher-leader at each participating school, with support from a Literacy Design Collaborative coach, provides local leadership 
for the professional learning community and coordinates learning community meetings. 


In Herman et al. (2015), three-fourths of teachers participated in common planning time, but only 26% did so every other 
week or more frequently. The authors do not describe the implementation experience of the sample of teachers in the other 
two studies that contribute to this intervention report (Wang et al., 2018, 2020). 


A Literacy Design Collaborative coach provides remote support to teachers implementing the program. During learning 
community meetings, coaches participate every other week by video conference to help teachers prepare instructional 
modules and tasks. Coaches may present course content related to Literacy Design Collaborative templates and tasks and 
direct teachers on how to access course content independently using CoreTools. In addition, coaches vet teachers’ newly 
developed instructional modules and tasks to assure their quality before they are used in the classroom. 


School and district instructional leaders may attend a coaching institute or coaching sessions, periodically participate in the 
learning community, and observe instruction and provide feedback to teachers. 


What Does Literacy Design Collaborative Cost? 


This preliminary list of costs is not designed to be 
exhaustive; rather, it provides educators an overview of the 


major resources needed to implement the Literacy Design 


Table 3. Cost ingredients for Literacy Design Collaborative 


Collaborative. The program costs described in Table 3 are 
based on the information available as of July 2020. 


Cost ingredients 


Description 


Source of funding 


Personnel 


Facilities 


Equipment and 
materials 


Classroom teachers implement Literacy Design Collaborative with their 
students. Teachers are supported in doing so through participating in a 
professional learning community and receiving remote coaching support. 
Schools must provide common planning time for teachers to participate in 
the learning community and schools may need to increase the total amount 
of planning time for participating teachers. No information is available how 
schools provided teachers with common planning time. 


Schools provide time for teachers to participate 

in common planning time for Literacy Design 
Collaborative activities. No information is available 
on how schools or school districts covered any 
costs associated with this additional planning time. 


Learning community meetings occur in a physical space within the school. 
Internet-connected computers are required to access intervention materials 
and remote coaching support. 


School districts or schools provide the meeting 
facilities and Internet access. 


Teachers obtain the materials needed to participate in Literacy Design 
Collaborative on the CoreTools online portal. Teachers can access these 
materials and the online professional learning community platform through a 
1-year license which costs $3,999 per school. 


The developer may have grant funding available 
to support the cost of the 1-year license fee. 


For More Information: 


About Literacy Design Collaborative 
90 Broad Street, 2nd Floor 
New York, NY 10004 
Attn: Literacy Design Collaborative 
Email: info@|dc.org Web: Idc.org. Phone: (212) 710-2781 


About the cost of the intervention 


Web: Idc.org 


Research Summary 


The WWC identified five studies that investigated the The WWC reports additional supplemental findings, such as 
effectiveness of Literacy Design Collaborative (Figure 1): those the study authors reported for a specific grade level, 
on the WWC website (https:/Awhatworks.ed.gov). These 
supplemental findings and findings from studies that either 


¢ Three studies meet WWC group design standards with 


Eeseryanus do not meet WWC standards or are ineligible for review do 
* Two studies do not meet WWC group design standards not contribute to the effectiveness ratings. 
The WWC reviews findings on the intervention’s effects The three studies of Literacy Design Collaborative that 
on eligible outcome domains from studies that meet meet WWC group design standards reported findings on 
standards, either with or without reservations. Based on this general literacy achievement and general social studies 
review, the WWC generates an effectiveness rating, which achievement. No other eligible findings were included 
summarizes how the intervention impacts, or changes, a among the studies that meet WWC group design standards.°* 
particular outcome domain. Citations for the five studies reviewed for this report are 


listed in the References section, on page 13. 


Figure 1. Effectiveness ratings for Literacy Design Collaborative 


0 studies meet WWC study meets WWC 2 studies do not 0 studies are 


standards without standards with meet WWC 
reservations reservations Jr late Fel ne by 


ineligible for 
review 


Contribute to effectiveness ratings Do not contribute to effectiveness ratings 


Literacy Design Collaborative has mixed effects on general literacy achievement 


Three studies that meet WWC group design standards with reservations contribute findings in the general literacy 
achievement domain. The WWC determined one study showed evidence of a positive and statistically significant effect 
of Literacy Design Collaborative on general literacy achievement (Herman et al., 2015), while two other studies showed 
evidence of indeterminate effects (Wang et al., 2018, 2020). 


Literacy Design Collaborative has no discernible effects on general social studies achievement 


The WWC determined that one study that meets WWC group design standards with reservations shows evidence of an 
indeterminate effect of Literacy Design Collaborative on general social studies achievement (Herman et al., 2015). 


Main Findings 


Table 4 shows the findings from the three studies of Literacy evidence of inconsistent effects on literacy achievement. The 
Design Collaborative that meet WWC standards. The table effectiveness rating for general social studies achievement 
includes WWC calculations of the mean difference, effect is no discernible effects, indicating no affirmative evidence 
size, and performance of the intervention group relative of effects on social studies achievement. These findings 

to the comparison group. Based on findings from the three are based on at least 31,003 students for general literacy 
studies that meet WWC standards, the effectiveness rating achievement and 19,962 students for general social studies 
for general literacy achievement is mixed effects, indicating achievement. 


Table 4. Findings by outcome domain from studies of Literacy Design Collaborative that meet WWC standards 


Mean 
(standard deviation) WWC calculations 
Sample Intervention Comparison Mean Effect Improvement 
Measure (study) Study sample size group group difference size index p-value 
Kentucky Grade 8 16,149 0.16 0.10 0.06 0.07 +3 <.01 
Performance Rating (0.93) (0.92) 
for Educational 
Progress (K-PREP) 
Reading? 
K-PREP Writing? Grade 8 13,972 nr nr 0.01 0.01 +1 63 
: : Statistically 
Outcome average for general literacy achievement (Herman et al., 2015) 0.04 +2 significant 
New York State Grades 4 and 5 468 -0.08 -0.02 -0.07 -0.07 3 >.05 
English Language (0.98) (1.02) 
Arts Assessment® 
New York State Grades 6-8 6,428 0.01 0.01 0.00 0.00 0 99 
English Language (1.00) (1.00) 
Arts Assessment® 
Not 
Outcome average for general literacy achievement (Wang et al., 2018) -0.03 -1 statistically 
significant 
Smarter Balanced Grades 4-8 7,958 0.04 -0.03 0.06 0.06 +2 >.05 
English Language (1.00) (1.00) 
Arts Assessment 
(Wang et al., 2020)° 
Outcome average for general literacy achievement across all studies 0.02 +1 
K-PREP Social Grade 8 19,962 0.09 0.10 0 0.00 0 90 
Studies (Herman (0.93) (0.92) 
et al., 2015)? 
Outcome average for general social studies achievement 0.00 0 


Notes: For mean difference and effect size values reported in the table, a positive number favors the intervention group and a negative number favors the comparison group. 
The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are given the 
intervention (measured in standard deviations of the outcome measure). An indicator of the effect of the intervention, the improvement index can be interpreted as the expected 
change in percentile rank for an average comparison group student if that student had received the intervention. For example, an improvement index of +1 means that the 
expected percentile rank of the average comparison group student would increase by 1 point if the student received instruction from a teacher who used Literacy Design 
Collaborative. A positive or negative improvement index does not necessarily mean the estimated effect is statistically significant. Some statistics may not sum as expected due 
to rounding. nr = not reported. 


2 For Herman et al. (2015), the authors provided unadjusted means and standard deviations for K-PREP Reading and Social Studies in response to an author query; means 
were not available for the K-PREP Writing outcome. This study is characterized as having a statistically significant positive effect on general literacy achievement because the 
estimated effect is positive and statistically significant. This study is characterized as having an indeterminate effect on general social studies achievement because the mean 
effect reported is not statistically significant. 


> Wang et al. (2018) required a correction for clustering and a difference-in-differences adjustment for the middle school analysis (grades 6-8). This was not required for the 
elementary school analysis because the student-level effect size was calculated from a statistical model with a clustering adjustment. The unadjusted means and standard 
deviations were provided in response to an author query. The p-value presented here for the middle school analysis was calculated by the WWC, because the p-value reported 
in the study was reported from an analysis that included an endogenous covariate (student time in core courses in the posttest year). The elementary school analysis did not 
include this endogenous covariate and so the p-value reported in the study is reported here. This study is characterized as having an indeterminate effect on general literacy 
achievement because the mean effect reported is not statistically significant. 

° For Wang et al. (2020), the authors provided unadjusted means and standard deviations in response to an author query. This study is characterized as having an 
indeterminate effect on general literacy achievement because the mean effect reported is not statistically significant. For more information, please refer to the WWC Procedures 
Handbook, version 4.0, page 22. 


In What Context Was Literacy Design Collaborative Studied? 


The following section provides information on the setting 

of the three studies of Literacy Design Collaborative that meet 
WWC standards, and a description of the participants 

in the research. This information can help educators 


WHERE THE STUDY WAS CONDUCTED 


Race 2% 
[i34% 


White Black Asian Other 


Free & Reduced-Price 
Lunch: 60% 


Special 


ig all 
- 3 studies, 34,816 students and 4,169 teachers in three States 


Kentucky, New York City, and a district in a West Coast state 


Education: 12% 


Grades 4-8 


understand the context in which the studies of Literacy 
Design Collaborative were conducted and determine whether 
the program might be suitable for their setting. 


Ethnicity 


Non-Hispanic Hispanic 


Gender: 49% Female 
51% Male 


Details of Each Study that Meets WWC Standards 


This section presents details for the studies of Literacy 
Design Collaborative that meet WWC standards. These 
details include the full study reference, findings description, 
findings summary, and description of study characteristics. 
Asummary of domain findings for each study is presented 
below, followed by a description of the study characteristics. 
These study-level details include contextual information 
about the study setting, methods, sample, intervention 
group, comparison group, outcomes, and implementation 
details. For additional information, readers should refer to 
the original studies. 


Research details for Herman et al. (2015) 


Herman, J. L., Epstein, S., Leon, S., Dai, Y., La Torre 
Matrundola, D., Reber, S., & Choi, K. (2015). The 


implementation and effects of the Literacy Design Collaborative 
(LDC): Early findings in eighth-grade history/social studies 

and science courses (CRESST Report 848). Los Angeles, 

CA: University of California, National Center for Research 

on Evaluation, Standards, and Student Testing (CRESST). 
Retrieved from http://cresst.org/wp-content/uploads/R848.pdf 


Findings from Herman et al. (2015) show evidence of a 
statistically significant positive effect of Literacy Design 
Collaborative on general literacy achievement (Table 5). 
The finding on general social studies achievement shows 
evidence of an indeterminate effect. The findings and 
research details summarized for this study come from 
two related citations, including the primary study listed 
above. See the References section on page 13 for a list of 
all related publications. 


Table 5. Summary of findings from Herman et al. (2015) 


Meets WWC Group Design Standards With Reservations 


Study findings 
Sample Average Improvement Statistically 
Outcome domain size effect size index significant 
General literacy achievement 634 teachers and 16,149 students 0.04 +2 Yes 
General social studies achievement 790 teachers and 19,962 students 0.00 0 No 


Table 6. Description of study characteristics for Herman et al. (2015) 


WWC evidence 
rating 


Setting 


Methods 


Study sample 


Intervention 
condition 


Comparison 
condition 


Outcomes and 
measurement 


Additional 
implementation 
details 


Meets WWC Group Design Standards With Reservations. This is a cluster quasi-experimental study that satisfies the 
baseline equivalence requirement for the individuals in the analytic intervention and comparison groups. For more information 


on how the WWC assigns study ratings, please see the WWC Procedures and Standards Handbooks (version 4.0) and 
\WWC Standards Briefs, available on the WWC website. 


The study took place in grade 8 history and science classrooms in Kentucky. Other details about the setting, such as the 
numbers of districts and schools, are not provided in the study. 


This study compares grade 8 students of teachers who received the intervention to other grade 8 students enrolled in 
similar history and science courses in the state during the 2012-13 school year. The study authors formed the comparison 
group by matching intervention group students to students in similar school and classroom settings in the state based on 
their grade 7 reading and science scores, gender, race/ethnicity, special education status, eligibility for free or reduced- 
price lunch, school Title | status, English learner status, whether they attended schools with similar prior effectiveness, and 
whether they were taught by teachers with similar prior effectiveness. The matching was conducted separately for each 
outcome measure. The same student might contribute test scores in reading, writing, and social studies, although not all 
students were successfully matched for all three outcome measures. 


The study included 16,149 students for the reading outcome, 13,972 students for the writing outcome, and 19,962 students 
for the social studies outcome. The intervention condition included 36 teachers of history or science. The comparison 
conditions included students linked to 598 teachers for the reading outcome and 754 teachers for the social studies 
outcome. The number of comparison teachers used for the writing outcome was not reported. Approximately 2% of students 
were Hispanic or Latino, 92% were non-Hispanic White, 4% were non-Hispanic Black, and 1% were non-Hispanic Asian. 
Approximately half the students were male, 0.3% were English learners, 47% were eligible for free or reduced-price lunch, 
and 9% were eligible for special education. 


Prior to the study year, intervention teachers had implemented the Literacy Design Collaborative program in their classroom 
for 1 or 2 years. During the study year, teachers were expected to participate in regularly scheduled collaborative planning 
time in a professional learning community and implement at least two 2- to 4-week literacy modules during the school year. 
Teachers were also expected to participate in two or three of the following types of professional development activities: 
classroom observations, coaching sessions, online courses, or meetings. Teachers may have participated in some of 

these activities during the professional learning community planning time. Although 73% of the 36 intervention teachers 
participated in regularly scheduled collaborative planning time, only 26% of the teachers did so at least every other week. 
Sixty-nine percent of the intervention teachers participated in any professional development during the school year. Those 
who participated received one to six professional development sessions. All intervention teachers taught at least one 
Literacy Design Collaborative module during the study year, nearly all taught two, and some taught as many as four. 


Students in the comparison condition were enrolled in similar history and science courses taught by teachers who were 
not participating in Literacy Design Collaborative. Further information on this business-as-usual comparison condition was 
not provided in the study. Comparison teachers may have participated in other business-as-usual training and professional 
development offered by their schools or school districts. 


The eligible outcomes for this study come from the 2013 end-of-year state assessment for grade 8, the Kentucky 
Performance Rating for Educational Progress. The study uses measures from the standardized assessments for reading 
and writing, which are in the general literacy domain, and for social studies, which is in the general social studies domain. 


The authors report that the amount and content of professional development, the number of participating teachers within 
schools, and other implementation factors varied across districts and schools. 


Research details for Wang et al. (2018) Evaluation, Standards, and Student Testing (CRESST). 
Retrieved from https: //eric.ed.gov/?id=ED600125 


Wang, J., Herman, J. L., Epstein, S., Leon, S., Haubner, J., La 
Torre, D., & Bozeman, V. (2018). Literacy Design Collaborative Findings from Wang et al. (2018) show evidence of an 
2016-2017 evaluation report for the New York City Department indeterminate effect of Literacy Design Collaborative on 
of Education (CRESST Report 856). Los Angeles, CA: general literacy achievement (Table 7). 

University of California, National Center for Research on 


Table 7. Summary of findings from Wang et al. (2018) 


Meets WWC Group Design Standards With Reservations 


Study findings 
Sample Average Improvement Statistically 
Outcome domain size effect size index significant 


General literacy achievement 1,641 teachers and 6,896 students -0.03 -1 No 


Table 8. Description of study characteristics for Wang et al. (2018) 


WWC evidence 
rating 


Setting 


Methods 


Study sample 


Intervention 
condition 


Comparison 
condition 


Outcomes and 
measurement 


Additional 
implementation 
details 


Meets WWC Group Design Standards With Reservations. This is a cluster quasi-experimental study that satisfies the 
baseline equivalence requirement for the individuals in the analytic intervention and comparison groups.. 


The study took place in 24 elementary schools and 105 middle schools in New York City with students in grades 4 through 8. 
The study included English language arts, social studies, and science classrooms in the middle schools. 


The study compares outcomes for students who received instruction from teachers participating in Literacy Design 
Collaborative with those who did not. The authors used two steps to match intervention students to similar comparison 
students. In the first matching step, the 25 intervention schools (5 elementary schools and 20 middle schools) were 
matched to 104 similar comparison schools (19 elementary schools and 85 middle schools) based on school grade span, 
the percentage of students eligible for free or reduced-price lunch, percentage of Black students, mean prior year student 
achievement in English language arts, and average teachers’ attendance rates and years of teaching experience. Students 
in the intervention schools were analyzed in the intervention condition if they received instruction from at least one Literacy 
Design Collaborative-trained teacher during the school year in which the outcome was measured. In the second matching 
step, each of the 3,448 intervention students (234 in elementary schools and 3,214 in middle schools) was matched to a 
similar student in a comparison school based on student grade, race/ethnicity, eligibility for free or reduced-price lunch, 
special education status, baseline score on English language arts and math tests, core classroom peers’ average score 

on the baseline English language arts test, and core classroom teachers’ average years of teaching experience. To be in 
the study sample in the intervention or comparison group, students were required to have both baseline and outcome test 
scores. Elementary students were required to be enrolled in the school for at least two-thirds of the school year. Middle 
school students were required to be enrolled in a core subject class for at least two-thirds of the school year. 


The 468 elementary school students—234 in each condition—were taught by 14 teachers in 5 schools in the intervention 
group and 100 teachers in 19 schools in the comparison group. The 6,428 middle school students—3,214 in each 
condition—were taught by 104 teachers in 20 schools in the intervention condition and 1,423 teachers in 85 schools in 

the comparison condition. Approximately half the students were male, 87% were eligible for free or reduced-price lunch, 
13% were English learners, and 24% were eligible for special education. Fifty-five percent of the students were Hispanic or 
Latino, 31% were non-Hispanic Black, 8% were non-Hispanic Asian, and 6% were non-Hispanic White. 


Intervention group schools began implementing Literacy Design Collaborative in the 2016-17 school year. Participating 
teachers were expected to develop at least one instructional module aligned with English language arts standards to use 

in their classroom in the school year, provide instruction using at least two modules per year, and participate in at least 

60 minutes of planning time in a professional learning community every 2 weeks. In addition, participating teachers were 
expected to receive feedback and support from a Literacy Design Collaborative coach remotely during learning community 
time, and through peer review comments on their instructional modules through the online CoreTools library. The authors do 
not describe the implementation experience of the sample of teachers in this study. 


Students in the comparison group were taught by teachers who did not participate in Literacy Design Collaborative. 
Comparison teachers may have participated in other business-as-usual training and professional development offered by 
their schools or school districts. 


The authors measured students’ scores on the 2017 New York State English Language Arts Assessment in grades 4 to 8, 
reporting findings separately for students in elementary and middle schools. This outcome measure is eligible for review in 
the general literacy achievement domain. 


Coaches worked directly with one or more teacher-leaders trained in each school to support implementation. Coaches 

and teacher-leaders worked together to structure learning community time and coaching support for other teachers in their 
schools. Literacy Design Collaborative staff also trained school administrators and district instructional specialists to support 
implementation, observe classroom instruction, and attend learning community sessions. 
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Research details for Wang et al. (2020) 


Wang, J., Herman, J. L., Epstein, S., Leon, S., La Torre, D., 
& Bozeman, V. (2020). Literacy Design Collaborative 2018- 
2019 evaluation report (CRESST Report 867). Los Angeles, 
CA: University of California, National Center for Research 


Table 9. Summary of findings from Wang et al. (2020) 


on Evaluation, Standards, and Student Testing (CRESST). 
Retrieved from https: //eric.ed.gov/?id=ED605029 


Findings from Wang et al. (2020) show evidence of an 
indeterminate effect of Literacy Design Collaborative in the 
general literacy achievement domain (Table 9). 


Meets WWC Group Design Standards With Reservations 


Study findings 
Sample Average Improvement Statistically 
Outcome domain size effect size index significant 
General literacy achievement 1,104 teachers and 7,958 students 0.06 +2 No 


il 


Table 10. Description of study characteristics for Wang et al. (2020) 


WWC evidence 
rating 


Setting 


Methods 


Study sample 


Intervention 
condition 


Comparison 
condition 


Outcomes and 
measurement 


Additional 
implementation 
details 


Meets WWC Group Design Standards With Reservations. This is a cluster quasi-experimental study that satisfies the 
baseline equivalence requirement for the individuals in the analytic intervention and comparison groups. 


This study took place in 137 elementary and middle schools in a large urban school district on the West Coast of the 
United States with students in grades 4 through 8. The study included English language arts, social studies, and science 
classrooms in the middle schools. 


This study compares outcomes for students who received instruction from teachers participating in Literacy Design 
Collaborative with those who did not. The authors used two steps to match intervention students to similar comparison 
students. In the first matching step, the 26 intervention schools were matched to 111 similar comparison schools from 

the same school district based on school grade span, the percentage of students eligible for free or reduced-price lunch, 
percentage of Black students, mean prior year student achievement in English language arts and math, and average 
teachers’ attendance rate and years of teaching experience. Students in the intervention schools were analyzed in the 
intervention condition if they received instruction from at least one Literacy Design Collaborative trained teacher during the 
school year in which the outcome was measured. In the second matching step, each intervention student was matched with 
a similar comparison student based on race/ethnicity, eligibility for free or reduced-price lunch, gender, baseline score on 
English language arts and math tests, English learner status, special education status, gifted student status, core classroom 
peers’ average achievement on the baseline English language arts test, and core classroom teachers’ average years of 
teaching experience. To be in the study sample in the intervention or comparison group, students were required to have 
demographic data, baseline and outcome test scores, and be enrolled for the entire school year. 


The 7,958 students in elementary and middle schools—3,979 in each condition—were taught by 89 teachers in 26 schools in 
the intervention group and 1,015 teachers in 111 schools in the comparison group. The sample breakdown between elementary 
and middle schools is not described. Approximately half the students were male, 69% were eligible for free or reduced-price 
lunch, 19% were English learners, and 8% were eligible for special education. Ninety-one percent of the students were 
Hispanic or Latino, 4% were non-Hispanic Black, 3% were non-Hispanic White, and 1% were non-Hispanic Asian. 


Intervention group schools began implementing Literacy Design Collaborative in the first cohort in the 2016-17 school 
year. A second cohort of schools participated starting in the 2017-18 school year. Participating teachers were expected 

to develop at least one instructional module aligned with English language arts standards to use in their classroom in the 
school year, provide instruction using at least two modules per year, and participate in at least 60 minutes of planning 

time in a professional learning community every 2 weeks. In addition, participating teachers were expected to receive 
feedback and support from a Literacy Design Collaborative coach remotely during learning community time, and through 
peer review comments on their instructional modules through the online CoreTools library. The authors do not describe the 
implementation experience of the sample of teachers in this study. 


Students in the comparison group were taught by teachers who did not participate in Literacy Design Collaborative. 
Comparison teachers may have participated in other business-as-usual training and professional development offered by 
their schools or school districts. 


The authors measure students’ English language arts scores in a pooled analysis across grades 4 to 8 and both cohorts of 
schools. Outcomes for schools participating in the first cohort and their matched comparison group schools were measured 
using the Spring 2018 Smarter Balanced English Language Arts Assessment. For the schools participating in the second 
cohort and their matched comparison group schools, outcomes were measured using the Spring 2019 Assessment. This 
outcome measure is eligible for review in the general literacy achievement domain. 


The study also reports supplemental findings separately by elementary and middle schools. Summaries of these 
findings are available on the WWC website (https://whatworks.ed.gov). The supplemental findings do not factor into the 
intervention’s rating of effectiveness. 


Coaches worked directly with one or more teacher-leaders trained in each school to support implementation. Coaches 

and teacher-leaders worked together to structure learning community time and coaching support for other teachers in their 
schools. Literacy Design Collaborative staff also trained school administrators and district instructional specialists to support 
implementation, observe classroom instruction, and attend learning community sessions. 
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Endnotes 


' See, for example, Goldhaber (2020) and Opper (2019). 


? Hanushek (2011) and Chetty et al. (2014), for example, 
describe differences across teachers in their impacts on 
academic achievement, and Jackson (2018) describes 
differences across teachers in their impacts on a variety of 
non-test score behaviors, including absences, suspensions, 
and course grades. 


3 The descriptive information for this intervention comes 
from the studies that contribute to this intervention report, 
and from the intervention developer. The What Works 
Clearinghouse (WWC) requests that developers review the 
intervention description sections for accuracy from their 
perspective. The WWC provided the developer with the 
intervention description in July 2020 and the WWC incor- 
porated feedback from the developer. Further verification 
of the accuracy of the descriptive information for this 
intervention is beyond the scope of this review. 


4The literature search reflects documents publicly available 
by October 2019. Reviews of the studies in this report used 
the standards from the WWC Procedures and Standards 
Handbook (version 4.0) and the Teacher Excellence topic 
area review protocol (version 4.0). 


5 The effects of Literacy Design Collaborative are not known 
for other outcome domains within the Teacher Excellence 
topic area, including general science achievement, general 
mathematics achievement, general achievement, English 
language proficiency, staying in school, progression in 
school, completing school, student social interaction, 
observed individual behavior, student emotional status, 
student engagement in school, instructional practice, 
teacher attendance, teacher retention at the school, 
teacher retention in the school district, teacher retention in 
the state, or teacher retention in the profession. 
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